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ABSTRACT 

The  problem  we  aim  to  solve  is  how  to  evaluate  the  dependability  of  software  at  the  stage  of 
architecture  definition.  Evidence,  such  as  the  process  maturity,  project  environment  and 
architecture  documentation  is  already  available  and  can  be  used  for  the  evaluation.  In  order  to 
create  a  holisticpictureof  the  state  of  dependability,  a  Bayesian  Network  (BN)  model  is  defined. 
The  paper  defines  a  quality  framework  which  guides  the  model  creation,  identifies  attributes 
characterising  dependability  and  presents  the  topology  of  the  model.  The  approach  to  the 
quantitative  definition  of  the  model  is  illustarted  by  examples.  The  model  is  aimed  to  help  with 
conducting  technical  risk  assessment  of  Airborne  Mission  Systems. 
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Evaluation  of  Software  Dependability  at  the 
A  rch  i  tectu  re  D  ef i  n  i  ti  on  Stage 


Executive  Summary 


Theachievementof  software  quality  goalsneeds  to  be  considered  during  all  stages  of  the 
softw  are  d  evel  opment  process.  Q  ual  i  ty  factors  are  effecti  vel  y  constrai  ned  by  the  arch  i  tectu  re 
and  therefore,  they  need  to  be  evaluated  as  early  as  possi  bleat  the  architecturedefiniti  on 
phase.  The  qual  ity  factor  we  ai  m  to  B/al  uate  is  dependabi  I ity  of  software. 

Thegoalsofsoftwarearchitecture  evaluation  at  AO  D/  Airborne  M  ission  Systems  areto:  (1) 
support  softw  are  systems  acquisiti  on;  (2)  assess  project  health  and  predi  ct  project  risks.  The 
model  described  in  this  report  aims  to  help  the  technical  risk  assessment  by  providing 
guidelines/ metrics  for  evaluating  the  quality  attributes  associated  with  software 
dependability. 

T 0  harness  the  process  of  software  dependabi  I  ity  eval  uati  on,  we  adopt  the  M  cCal  I  qual  ity 
model  with  some  modification.  The  model  represents  a  four  level  hierarchy  of  quality 
factors,  Le/elland  Le/el  2  quality  attributes  and  metrics. 

We  aim  to  derive  an  estimate  for  each  quality  factor,  where  the  quality  factors  defining 
dependability  are  Rd 'lability,  Safety  and  M  aintainability.  In  order  to  achieve  this,  webuild  a 
"dependency  graph"  between  the  qual  ity  factors  (thefi  rst  level  of  the  qual  ity  framework), 
level  1  qual  ity  attributes  (testability,  complexity,  openness,  robustness,  modularity  etc),  and 
level  2  attributes  characterising  the  engineering  process,  the  organisation  and  the  softw  are 
architecture  (process  maturity,  management  practices,  team  experience,  quality  of 
requirements  elicitation,  architecture  style,  coupling,  error  propagation,  capacity  margin 
etc).  Finally,  checklists  are  formulated  to  reason  about  the  qual  ity  attributes. 

Bayesian  Networks(BN)  provideamechanism  to  expressthecausal  relationships  between 
the  elements  of  the  quality  framework.  Bayesian  Network  is  a  graphical  model  that 
represents  the  dependency  between  the  model 's  vari  abl  es;  these  vari  abl  es  represent  our 
quality  attributes  and  quality  factors.  These  relationships  are  causal,  but  our 
understanding  of  them  is  not  complete,  which  is  why  it  is  possible  to  describe  them 
probabilistically. 

This  report  describes  the  different  attributes  forming  the  Bayesian  model  to  be  used  for 
evaluation  of  software  dependabi  I  ity:  nodes  (representing  the  qual  ity  factors  and 
attributes)  and  Conditional  ProbabilitiesTables  (representing  the  probability  given  node 
taking  each  of  its  values).  The  end  goal  is  to  provide  a  tool  supporting  the  technical  risk 
assessment  activities  of  mission  system  softw  are  in  Defence  acquisition  projects. 
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1.  Introduction 

1.1  Software  Architecture  and  Quality  of  Software 

Software  is  the  critical  enabling  technology  for  consumer  electronics  as  well  as  for  security 
and  safety  critical  applications  used  in  avionics,  space,  railway  and  transport,  process  control 
and  medical  systems.  This  trend  finds  its  expression  in  monetary  terms;  for  example  the  UK 
Defence  Procurement  Agency  spends  over  £lBn  of  its  £7Bn  annual  budget  on  software,  and 
the  percentage  spent  on  software  will  only  grow  [MoD  2006].  The  increasing  rolesoftware 
playsin  modern  day  technology  puts  software  quality  in  the  limelight. 

The  term  software  quality  is  a  generic  term  which  needs  a  general  consensus  on  meaning 
[Voas2008].  It  represents  the  notion  of  software  being  fit  for  a  purpose,  i.e.  the  things  the 
user/  customer  is  expecting  from  the  product.  These  expected  things  are  not  functional 
requirements  -  implementing  the  functional  requirements  is  well  defined.  It  has  been  a 
number  of  years  since  it  was  pointed  out  that  the  non-functional  requirements  (N  FR),  also 
known  as"ilities",  are  the  ones  which  are  the  challenge  and  lead  to  the  blow-up  in  costand 
schedule.  Functionality  and  quality  attributes areorthogonal,becauseotherwisethechoice of 
afunction  would  define  the  level  of  a  certain  quality  attribute[Bassetal.  2003]. 

To  harness theprocess of  softwaredevelopmenttoward  achieving  manageablequality  goals, 
these  goals  had  to  be  measured,  and,  as  a  result,  quality  measurement  frameworks  were 
proposed .  The  M  cCal  I  qual  ity  model  (i  1 1  ustrated  i  n  Fi  gure  1)  is  based  on  three  types  of  qual  ity 
characteristics:  Quality  Factors,  Quality  Criteria  and  metrics  [McCall  1994]. 


Figure  1:  Q  uality  measurement  framework 


A  quality  factor  (also  called  quality  characteristic  in  ISQ/ 1  EC  9126  [I  EC  9126])  is  a  quality  goal 
and  represents  a  characteristic  of  the  software  that  a  customer  would  relate  to  the  overall 
quality.  Consequently,  thischaracteristicreflectstheexternal  (user)  point  of  view  and  would 
typically  be  given  in  the  requirements.  Some  studies  apply  the  term  quality  attribute 
[Florentz  etal.  2006]  for  this  characteristic,  which  introduces  some  confusion. 
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The  second  level  of  the  software  quality  framework  -  quality  criteria,  represents/  provides 
software  prod uct  attri  butes  rel  ated  to  the  qual  i  ty  factors.  These  attri  butes  refi  ect  the  i  nternal 
(developer)  point  of  view.  The  third  le/el  of  the  quality  framework  consists  of  metrics 
associated  with  the  criteria.  These  metrics  define  whether  the  criteria  exists  and  to  what 
degree.  They  can  be  checklists  or  review/  inspection  guidelines  which  "grade"  the  quality 
criteria  or  quantitative  measures  of  characteristics  such  as  size,  complexity  etc. 

The  quality  framework  provides  a  basisfor  a  disciplined  approach  to  assess  softwarequality. 
Thelogical  steps  to  beundertaken  within  such  a  goal-oriented  approach  are: 

•  Identify  quality  factors  we  are  interested  in; 

•  Consider  what  criteria  impact  these  quality  factors; 

•  Consi der  the i  nterdependenci es  between  thefactors  (common  cri teri a  w hi ch  may  have 
positive  impact  on  onefactor  while  having  negative  impact  on  another  factor); 

•  P  rovi  d  e  measu  rements  ( metr  i  cs  p  rog  ram)  to  assu  re  the  q  u  al  i  ty  cr  i  ter  i  a  i  s  bu  i  1 1  i  nto  th  e 
software  product;  and 

•  Track,  analyseand  improve  metrics  collection. 

Theachi  evement  of  the  software  qual  ity  goals  needs  to  be  consi  dered  duri  ng  al  I  stages  of  the 
software  devel  opment  I  ife  cycl  e.  We  are  i  nterested  i  n  the  earl  y  phases  of  the  devel  opment  - 
up  to  and  including  the  software  architecturedefinition  phasewhich  is  part  of  the  software 
design.  Software  design  fits  between  the  software  requirements  analysis  and  software 
construction  and  isconsidered  atwo step  activity:  architectural  (or  high  level  design)  design 
and  detailed  design  [IEEE/  EIA  12207],  [SWEBOK  2004].  Architectural  design  describes  how 
software  is  decomposed  and  organised  into  components  (the software  architecture). 

Softw  are  archi  tectu  re  i  s  at  the  centre  stage  i  n  mod  ern  softw  are  engi  neeri  ng  as  the  pi  atform  for 
making  major  decisions  which  will  have  long  term  impact  on  all  consequent  artefacts,  like 
mapping  functional  ity  to  hardware  and  software,  configuration  of  components,  breakdown  of 
interfaces  etc.  Although  architecture  by  itself  is  not  ableto  achieve  qual  ity,  it  provides  the 
basisfor  achievi  ng  qual  ity  [Bass  et  al .  2003].  Qual  ity  factors  are  effectively  constrai  ned  by  the 
architecture  and  therefore,  they  need  to  be  evaluated  at  the  architecture  level. 

The  goals  of  software  architecture  evaluation  for  Airborne  Mission  Systems  (AMS)  are  to: 
(1)  support  software  systems  acquisition;  (2)  assess  project  health  and  predict  project  risks. 
The  assumed  relations  between  design  solutions  and  quality  requirements  are  not  always 
correct  [Bosch  etal.  2001].  Detailed  evaluation  giving  sufficient  insight  in  the  attri  butes  of  an 
architecturedesign  isexpensive,  consuming  consi  derableti  me  and  resources.  The  model  we 
de/elop  aims  to  help  the  technical  risk  assessment  by  providing  guidelines/  metrics  for 
B/al  uati  ng  thequal  i  ty  attri  butes  associ  ated  with  dependabi  I  ity,  as  dependabi  I  ity  is  of  hi  ghest 
priority  in  avionics  systems. 

1.2  Dependability 

This  report  isconcerned  with  theidentifi  cation  and  evaluation  of  thedependabi  I  ity  factorsfor 
Airborne  Mission  Systems.  Dependability  is  the  ability  of  a  system  to  avoid  service  failures 
whi  ch  are  more  frequent  and  more  severe  than  expected  [A  vizi  enis  et  al .  2004].  Dependabi  I  ity 
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is  a  generic  term  which  includes  several  quality  factors:  reliability,  availability, 
maintainability,  safety,  and  integrity.  This  report  builds  on  [Uzunov  et  al.  2008],  where  a 
survey  was  undertaken  to  review  the  existing  approaches  to  dependability  assessment  and 
i  mprovement  at  vari  ous  stages  of  the  software  devel  opment  I  ife  cycl  e  (SDLC). 

Predicting  dependability  of  such  complex  computer  systems  is  difficult  because  of  [Prasad 
1998]: 

•  Problems  of  integration  resulting  from  unexpected  interaction  between  the  large 
number  of  components; 

•  Multi  pie  attributes  characterising  dependability; 

•  The  sci  enti f i  c  pri  nci  pi  es  of  measu rement  have  not  been  appi  i  ed  w  i  th  ad equ ate  r i  gou  r 
for  decision  making  during  the  earlier  stages  of  the  devel  opment  process. 

The  ultimate  questions  we  seek  to  answer  are:  (1)  What  quality  criteria,  project  and 
architecturecharacteristicscontributetothedependability  factor  and  how  do  they  interact;  (2) 
H  ow  these  characteristics  can  be  captured/  measured  for  a  specific  project;  (3)  H  ow  can  we 
use  these  characteristics  to  predict  the  risk/  qual  ity  of  a  specific  project. 

Because  software  architecture  has  critical  impact  on  the  quality  factors  (including 
dependability),  weconcentrateonthearchitecturelevel  evaluation  and  issues  stemming  from 
th e softw aredevelopmentduringrequi rements d ef i n i ti on  an d  arch i tectu re d ef i n i ti on .  I  n th i s 
report  we  discuss  the  quality  factors,  criteria  and  attributes  impacting  dependability,  their 
interrelations  and  propose  a  model  to  be  developed  based  on  this.  Chapter  2  lists  the 
approaches  used  for  software  architecture  evaluation,  elaborates  the  characteristics  of  the 
Airborne  Mission  Systems  influencing  software  dependability  and  presents  the  reasons  for 
applying  a  Bayesian  Network  (BN)  model  to  the  evaluation  of  software  dependability. 
Chapter  3  descri  bes  the  Bayesi  an  N  etwork  model  -  thesemanti  cs  of  the  vari  abl  es  represented 
as  nodes,  the  dependencies  between  nodesand  descri  bes  how  the  Conditional  Probabilities 
Tables  are  defined.  Chapter  4 concludes. 


2.  Softw  are  Architecture  Evaluation 

2.1  Summary  of  Approaches 

The  architecture  evaluation  techniques  can  begrouped  in:  (1)  architectureoriented  techniques 
and  (2)  quality  attribute  focused  techniques  [Bosch  et  al.  2001]. 

The  architecture  oriented  techniques  are  built  around  expert  reviews  following  defined 
guidelines.  Typically,  the  reviews  are  held  after  the  architecture  is  defined.  Most  of  these 
techniques  are  briefly  described  in  [Bergner  et  al.  2005].  [Babar  et  al.  2004]  proposes  a 
framework  for  their  comparison  and  assessment.  Here  are  some  examples  of  architecture 
oriented  approaches: 

•  Software  Architecture  Analysis  Method  (SAAM)  -  Evaluation  of  software 
architectures  is  executed  through  scenarios,  quality  attributes  and  quality  objectives. 
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•  Architecture  Tradeoff  Analysis  method  (ATAM)  -  This  method  concentrates  on 
identifying  critical  tradeoff  points  which  will  impact  the  architecture  (including  its 
quality  atributes)  and  the  risk  involved.  Asa  result,  the  ad  vantages  and  disadvantages 
of  each  tradeoff  are  well  understood. 

•  Cost-Benefit  Analysis  Method  (CBAM)  -  builds  up  on  ATAM  by  adding  cost  of 
architectural  decisisons. 

•  Architecture  Tradeoff  Analysis  Method  (ALMA). 

•  A  cti  ve  Revi  ew  f or  I  ntermi  d  i  ate  Desi  gns  (A  Rl  D ) . 

•  SoftwareArchitectureComparison  Analysis  Method  (SACAM). 

•  QUASAR  [Firesmith  2006]  -  QUASAR  system  architecture  assessment  method  is 
essentiallyan  auditfollowing  theCMU  SEI  QUASAR  methodology  and  isbased  upon 
quality  cases.  A  quality  case  is  a  generalisation  of  a  safety  case  consisting  of  claim, 
argument  and  evidence.  The  audit  looks  at  whether  the  presented  evidence  proves 
that  the  architectural  decisions  and  rationales  lead  to  software  architectures  ableto 
support  the  qual  ity  requi  rements. 

•  Quality  SoftwareArchitectureReview  -Thismethod  defines  a  process  and  guidelines 
for  an  architecture  review  by  external  reviewers. 

•  Domain  Specific  SoftwareArchitectureComparison  Model  (DoSAM)  [Bergner  etal. 
2005].  It  is  based  on  quality  attributes  and  scenarios,  but  is  tailored  for  the  needs  of  a 
domain  analysis.  It  introduces  architectural  services  for  the  purpose  of  an  abstratct 
description  of  the  application  domain. 

The  quality  attribute  focused  techniques  aim  to  come  up  with  an  estimate  for  each  quality 
attri  bute  or  compare  architectures  against  a  given  quality  attribute.  These  techniques  (which 
may  be  based  on  quantitative  or  qualitative  assessment)  can  be  grouped  further  into  3+1 
methods:  scenario-based,  simulation,  mathematical  modeling/  metrics  and  experience-based 
reasoning  [Bosch  2001].  The  brief  descriptions  of  the  process  to  be  followed  for  each 
techniques  is  taken  from  [Lundberg  etal.  1999]. 

•  Scenario  based  evaluation  -  a  profile  for  a  specific  quality  attribute  is  created  (a 
scenario  profile  is  a  set  of  typical  scenarios,  e.g.  hazard  scenariosfor  safety).  Then  the 
i  mpact  of  the  scenari  ons  on  the  archi  tectu  re  i  s  assessed .  Based  on  thi  s,  a  pred  i  cti  on  can 
be  made  about  the  quality  attribute.  Such  a  technique  for  assessing  the  optimal 
maintainability  is  described  in  [Bosch  etal.  2001]. 

•  Si  mu  I  ati  on  -  the  arch  i  tectu  re  i  s  mod  el  I  ed  (usi  ng  arch  i  tectu  re  d  escri  pti  on  I  anguages  or 
conventional  languages)  and  the  results  from  runing  scenarios  on  the  model  are 
anal ysed  i  n  order  to  pred i  ct  the  qual  i  ty  attri  bute  u nder  i  nvesti  gati  on .  A  n  exampi  e  of 
such  an  approach  is  [Gregoriades  et  al .  2005]  where  the  system  is  descri  bed  usi  ng  the 
i*  language  and  then  scenarios  are  created  based  on  interviews  with  the  users  and 
stakehol  ders.  The  scenari  os  (I  ists  of  I  i  nked  tasks)  are  converted  i  nto  executabi  e  form 
and  a  Bayesian  Network  model  isplugged  intothetool.Themodel  takes  as  inputs  the 
characteristics  of  each  task  and  the envi ronment  and  calculatesthe  reliability  for  the 
given  scenario. 

•  Mathematical  modeling  (including  metrics) -thearchitectureis presented  intermsof 
an  appropriate  mathematical  model;  the  model  output  is  calculated  and  interpreted  in 
orderto  predict  thequality  attri  bute.  Thisapproach  includesthecollection  of  product 


4 


DSrO-TR-2428 


and/  or  process  metrics  which  are  used  to  make  a  prediction  about  a  given  quality 
attribute.  Many  studies  are  dedicated  to  the  definition  of  various  metrics  and  the 
val  i  dati  on  of  these  metri  cs  as  predi  ctors  of  specifi  c  qual  iti  es.  [Shereshevsky  et  al .  2001] 
defines  coupling  and  cohesion  metrics  characterising  information  flows  in  the 
architecture.  These  metrics  are  related  to  quality  criteria  like  error  propagation  and 
change  propagation.  Design  metrics  reflecting  connectivity  of  architecture  components 
and  the  information  in/  outflowsaswell  as  component's  internal  structure  are  defined 
in  [Stineburg  etal.  2005].  The  referenced  report  further  studies  the  application  of  the 
design  metrics  for  evaluating  software  reliability. 

•  Experience-based  assessment  is  based  on  theexperienceofthedesignerswho  review 
the  architecture  looking  for  weaknesses  against  a  given  quality  attribute. 

Our  approach  to  eval  uati  ng  the  dependabi  I  i  ty  attri  butes  of  A  i  rborne  M  i  ssi  on  Systems  may  be 
assigned  formal  ly  to  thegroup  of  methods  based  on  mathematical  modeling.  Weare  working 
on  a  Bayesian  Network  model  called  AMS-BN  (Air  Mission  System- Bayesian  Network). 

2.2  Applying  Bayesian  Networks  to  Dependability 

The  only  means  of  direct  evaluation  of  dependability  is  through  operational  testing  of  the 
software.  In  addition  to  the  testing,  other  information  is  available  for  evaluation,  which 
characterises  the  devel  opment  organi  sati  on,  the  qual  ity  of  the  software  engi  neeri  ng  process 
applied  during  the  Software  Development  Life  Cycle  and  the  engineering.  Some  of  this 
i  nformati  on  i  s  pubi  i  cl  y  know  n,  e.g.  w  hether  theorgani  sati  on  has  been  assessed  agai  nst  CM  M I 
or  I  SO 2000  or  some  other  standard.  Applicable  standards  can  be  identified  from  the 
requ  i  rements  d  ocu  mentati  on.Theprojectpl  ans  w  ou  I  d  provi  de  i  nformati  on  about  the  sel  ected 
development  process,  thecollection  of  metrics  and  would  indicatewhethertheorganisation 
really  functions  at  the  level  of  maturity  at  which  it  was  assessed.  Finally,  most  of  the 
information  concerning  the  engineering  activities  can  be  obtained  as  a  result  of  an  audit  or 
request  for  i  nformati  on  to  the  devel  oper.  I  n  the  case  of  the  I  atter,  the  i  nformati  on  needs  to  be 
identified  and  included  as  part  of  thecontract  negotiation. Thisinformati  on  may  include  high- 
level  design  decisions  as  reflected  in  thesoftwarearchitecturedocumentation,  evidence  for 
and  results  from  reviews,  safety  analysis.  Failure  and  Effect  Anal  ysi  sand  code  inspections. 

None  of  the  evidence  mentioned  above  can  alone  provide  enough  information  about  the 
dependability  characteristics  of  software.  Probably  the  most  important  thing  for 
understanding  the  software  reliability  measurement  is  experience  and  judgement.  Our 
conclusion  is  that  the  only  way  to  build  a  holistic  picture  of  the  state  of  dependability  is  to 
develop  a  comprehensive  framework,  where  all  the  quantitative  and  qualitative  evidence 
described  inthisreport(people,  process  and  product)  iscaptured.  Such  a  framework  needsto 
provi  de  i  nput,  gu  i  dance  and/  or  suggesti  ons  to  the  expert  to  make  conci  usi  ons.  The  sel  ected 
approach  should  be  able  to  capture  uncertainty,  expert  judgement  and  incomplete 
information.  Such  a  framework  can  be  used  for  assessment  of  dependability  and  consequently 
to  help  Technical  Risk  Assessment  (TRA). 

Various  approaches  are  used  to  combi  nedisparateevidencein  order  to  make  a  valuation  of 
the  overall  dependability  of  a  system.  The  DATUM  (Dependability  Assessment  of  Safety 
Critical  Systems  through  the  Unification  of  Measurable  Evidence)  project  in  the  UK  (the 
Centrefor  Software  Reliability,  City  University,  London)  investigated  several  formal  isms  used 
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to  model  uncertainty  [Falla  1998].  The  methods  considered  were  Bayesian  probability, 
Dempster- Shafer  theory  of  belief  functions,  fuzzy  sets  and  possibility  theory.  Although  no 
single  formal  ism  for  uncertainty  was  found  perfect,  Bayesian  probability  was  chosen  as  the 
most  mature  and  well  developed  formal  ism  at  the  ti  me  [  Fenton  etal.  1998].  The  obstacle  for 
usi  ng  Bayesi  an  probabi  I  ity  i  n  cases  of  multi  pi  eevi  dence  wasthecompi  ex  computati  ons.  Thi  s 
probi  em  has  been  mosti  y  overcome  w i  th  the  d evel  opment  of  al  gori  thms,  sol  uti  ons  and  tool  s 
in  support  of  this  formalism  -  the  Bayesian  Networks.  The  developments  in  network 
propagation  algorithms  make  Bayesian  inference  computationally  feasible  for  solving 
complex  problems.  Bayesian  inference  can  also  be  used  for  "what  if"  analysis.  Brief  reviews 
and  detaiisfor  most  of  the  avail  able  Bayesi  an  Model  ling  tool  scan  befound  in  [Anthony  2006] 
and  [M  urphy  2005].  1 1  is  i  nteresti  ng  to  note  i  n  this  context  the  paper  [Si  mon  et  al .  2006],  where 
Bayesian  Network  implementation  of  the  Dempster- Shafer  theory  is  used  to  model  the 
reliability  uncertainty.  [Ziv  et  al.  1997]  also  identified  Bayesian  Networks  as  a  suitable 
technique  for  modelling  uncertainty  in  software  systems. 

Bayesian  Networks  were  also  used  bytheFASGEP  (Fault  AnalysisoftheSoftwareGenerati  on 
Process)  project  to  determine  the  fault  propensity  of  software  processes  [Falla  1998].  The 
direction  taken  by  the  DATUM  project  has  being  foil  owed  in  a  series  of  projects  within  the 
RA  D  A  R  ( Ri  sk  A  ssessment  and  D  eci  si  on  A  n  al  ysi  s  Research  )GroupinQueenMaryUni  versi  ty 
of  London  led  by  N  .Fenton.  [Wang  et  al.  2006]  reported  on  the  application  of  BN  for  project 
level  estimation,  where  the  accent  is  on  the  development  and  the  test  phases  of  theSDLC. 
[Perez-M  inana  etal.  2006]  reported  on  the  development  of  BN  models  for  prediction  of  fault 
insertion  and  fault  removal.  The  models  were  used  as  part  of  the  software  development 
process  in  Motorola,  Toulouse.  Itisworth  noting  that  theaccuracy  of  theinitial  predictions  of 
the  generic  models  required  a  calibration  based  on  measures  from  concrete  projects.  A 
procedure  is  suggested  to  cal  ibratethe  models  and  arrive  at  an  improved  BN .  Oneapproach 
wasto  vary  the  values  associated  with  each  nodethat  are  used  asinputstotheintermediate 
nodes.  The  other  approach  (which  produced  better  results)  used  linear  regression  and 
Principal  Component  Analysisto  build  the  intermediate  and  the  output  nodes. 

BN  models  have  been  used  for  scenario- based  analysis  and  assessment  of  Non-Functional 
Requirements  (including  reliability)  [Sutcliffe  etal.  2002]  and  [Gregoriadesetal.  2005].  The 
models  are  pi  ugged  i  nto  the  System  Requi  rements  A  nalyser  tool .  Scenari  os  aredesi  gned  and 
depending  on  the  selected  tasks,  the  technology  attributes,  the  human  attributes  and  the 
environment  variables,  the  BN  model  provides  evaluation  of  the  reliability  for  the  scenario. 

Our  research  was  i  nspi  red  by  the  approach  descri  bed  i  n  [Fenton  et  al .  2007]  and  also  stems 
from  [Gurp  2003],  where  a  Baysean  Net  model  for  reasoning  about  software  architecture 
attributes  is  developed.  The  model  uses  a  hierarchy  of  quality  factors,  quality  criteria  and 
architecture  attributes.  The  major  differences  to  our  model,  considering  our  specifc  focus  on 
Airborne  Mission  Systems  (AMS),  isthatadifferentsetof  quality  criteria  and  architecture 
attri butes  have  been  identified.  Our  model  also  includes  characteristics  associated  with  the 
process  and  the  project,  since  these  have  a  major  impact  on  the  dependability  of  mission 
cri  ti  cal  systems.  Ourworkgoesfurtherinthequality  f  ramew  or  k  by  i  d  enti  fy  i  ng  metr  i  cs  ( i  n  th  e 
form  of  guidelines)  for  qualitativeassessement  of  some  criteria,  which  are  domain  specific. 
Another  analogue  to  our  work  can  be  seen  in  [Florentz  et  al.  2006],  where  the  same 
fundamental  approach  to  quality  evaluation  is  observed,  namely,  a  hierarchy  of  quality 
factors  and  quality  criteria  is  identified,  and  relationships  between  architectural  elements  and 
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qualityfactorsareinvestigated.Thequotedtechniqueusesapragmaticand  intuitive  method 
of  assigning  quantitativevaluesto  the  quality  factors,  but  in  comparison,  our  model  offers 
more  flexibility  (e.g.  Bayesian  inferenceallows"whatif"  analysis). 

We  are  i  nterested  i  n  the  rel  ati  onsh  i  ps  betw  een  the  archi  tectu  re  and  organ  i  sati  on  attri  butes  on 
one  hand,  and  the  dependability  factors.  These  relationships  are  causal,  but  our 
understanding  is  not  complete  (or  is  uncertain),  which  is  why  we  can  describe  them 
probabilistically.  A  Bayesian  Network  is  a  type  of  causal  model  which  uses  Bayesian 
probabi  I  ity.  Bayesi  an  N  etwork  is  a  graph!  cal  model  -  a  di  rected  acycl  i  c  graph,  whi  ch  captures 
probabilistic  relationships  between  the  model's  variables;  these  variables  represent  our 
attri  butes  and  qual  ity  factors.  The  vari  abl  es  are  represented  by  nodes,  whi  I  ethearcs  (I  i  nks)  of 
the  BN  represent  cond  i  ti  onal  i  nterdepend  end  es  betw  een  the  vari  abl  es.  N  od  es  w  i  thout  parents 
arecalled  root  nodes;  they  have  a  prior  distribution  associated  with  them.  In  our  model  most 
of  the  archi  tectu  re  and  organisation  attri  butes  are  rep  resented  by  root  nodes.  Every  nodewith 
parents  is  associated  with  a  Conditional  Probabilities  Table  (CPT)  that  represents  the 
probabilities  of  that  node  taking  each  of  its  values,  given  the  combi  nations  of  values  of  its 
parent  nodes. 

The  general  steps  in  constructing  a  BN  aredescribed  in  [Korbetal.  2004].  Firstthetopology  of 
the  net  is  defined  by  identifying  the  variables -often  starting  from  the  root  causes  and  adding 
new  variables  until  the  leaves  of  the  net  are  reached.  One  node  per  variable  is  entered,  the 
nodes  are  connected  and  thei  r  names  and  states  aredefi  ned .  The  next  step  i  s  the  defi  niti  on  of 
dependencies  between  the  nodes,  i.e.  filling  the  CPT.  When  the  net  is  constructed,  it  can  be 
applied  to  a  specific  case  by  entering  the  known  values  of  the  variables  (i.e.  evidence)  into 
their  corresponding  nodes.A  BN  tool  will  dofor  us  a  probabilistic  inference  in  order  to  find 
new  beliefs  (posterior  probabilities)  for  all  other  variables. 

Bayes'  theorem  is  used  asa  basisfor  updating  the  information  in  a  BN  .Thetheorem  calculates 
the  probabi  I  ity  of  two  (or  more)  dependent  events  A  and  B: 

P(A,  B)  =  P(B,  A)  =  (P(B  I  A)  *  p(A))  /  p(B) , 

where  p(A|  B)  isthe  probability  of  eventBhappening  given  A  has  already  happened.  Bayes' 
theorem  may  be  too  complex  if  an  exact  solution  is  required  for  a  large  number  of  events, 
however,  for  specific  classes  it  can  be  efficiently  solved.  Many  algorithms  have  also  been 
developed  for  finding  approximate  solutions  of  the  conditional  probabilities  in  a  Bayesian 
Network  [Charniak  1991].  OncetheCPT  isfilled  in,  the  Bayesi  an  Network  ensures  that  the 
numberswill  be  consistent  and  the  network  will  uniquely  definea  distribution. 

Our  beliefs  in  the  software  dependability  factors  are  influenced  by  many  attributes  as 
illustrated  in  Figures.  BN  saccommodatefor  the  combi  nation  of  different  variables:  some  of 
them  precisely  defi  ned  (e.g.  defect  contai  nmentfor  a  specifi  c  phase)  or  somebei  ng  eval  uated 
qualitatively  based  on  expert  opinion  or  guidelines  developed  by  experts.  BNs  provide  a 
mechanism  for  combining  the  evidence  from  architecture  and  organisational  attributes  in 
order  to  calculate  the  probability  that  the  dependability  factors  have  a  certain  value.  The 
network  is  updated  as  soon  as  new  evidence  is  entered. 
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2.3  C  haracteri sties  of  the  AMS  domain 

An  AMS  is  employed  in  a  specific  environment  which  imposes  additional  constraints  or 
requirements  in  comparison  to  a  general  purpose  computer  system.  Typically,  the 
development  isa  collaborativeeffort  of  multi  pie  partners  and  subcontractors  with  different 
culture  as  well  as  widely  geographically  dispersed  teams.  Well  defined  standards  for 
documentation,  interfaces  and  protocols  are  essential  inthiscase.AMSare  mission  and  safety 
critical,  hencethematurity  ofthedevelopmentprocess  and  experience  of  theorganisati  on  are 
a  primary  concern. 

A  i  r  pi  atforms  (and  thei  r  A  M  S)  are  i  n  servi  ce  for  I  ong  peri  ods  of  ti  me  i  n  the  range  of  tens  of 
years.  Asa  resu  1 1,  an  enormous  probi  em  i  s  that  of  unavai  I  abl  e  el  ectroni  c  parts  and  the  agei  ng 
of  technologies.  As  pointed  in  [Sandborn  2008],  an  estimated  3%  of  the  global  pool  of 
electronic  components  is  discontinued  each  month.  Electronic  parts  obsolescence  -  also 
known  as  Di  mi  nishi  ng  M  anufacturi  ng  Sources  (DM  S)  or  M  anufacturi  ng  Sources  and  M  ateri  al 
Shortages  (DM  SM  S)  i  s  an  issue,  si  nee  the  majority  of  processor  and  memory  chi  ps  i  n  A  M  S  are 
COTS  units.  In  order  to  keep  the  cost  of  technical  refreshes  and  upgrades  under  control, 
architecture  should  havecharacteristieslikemaintainability,  scalability  and  modifiability. 

The  development  cycle  for  AMS  is  longer  than  the  current  technology  cycle  and  leads  to 
technol  ogy  agei  ng.  Thi  s  and  the  I  ong  I  ifeti  me  of  a  pi  atform,  mean  that  the  i  nsati  abl  e  hu  nger 
for  space,  power  and  weight  on  an  aircraft  platform  will  result  in  requirements  for  new 
capability  or  substitution  of  existing  blocks  with  new  ones  offering  improved  capability.  This 
means  also  that  spare  processor  capacity,  bus  bandwidth  and  PCB  spaces  have  to  be  provided, 
together  with  an  architecture  which  has  to  be  scalable  in  order  to  incorporate  the  increased 
capacity.  Another  prerequisitefor  futureupgradesisarchitectureopenness  (both  for  software 
and  hardware),  i.e.  the  architecture  has  to  be  based  on  publicly  available  widely  accepted 
standards. 

Changes  in  theunderlyinghardwarego  hand  in  hand  with  thenecessityto  port  or  rewritethe 
associated  software.  Thus  upgrades  and  technology  refreshes  require  appropriate 
partitioning  of  the  software  and  mapping  to  hardware  so  that  any  changes  will  not  cause 
unexpected  changes  in  other  parts  of  the  system.  Appropriate  portioning  atdifferent  levels 
can  enable  incremental  integration  which  decreases  the  integration  risks  compared  with  the 
big-bang  approach.  The  selection  of  an  appropriate  architecture  pattern  will  help  isolate 
changes  in  the  hardware  to  a  specific  software  block. 

Airborne  Mission  Systems  (as  well  as  any  new  military  systems)  are  characterised  by  growing 
complexity  and  size.  Thehi  gh  complexity  of  thesystems  is  madeeven  more  complex  with  the 
trend  toward  i  nteroperabi  I  ity  of  systems  w  hi  ch  have  not  been  desi  gned  to  tal  k  to  each  other. 
The  reliance  of  operations  on  information  processing  and  networksledtotherequirementfor 
all  USDoD  acquisition  programsto  address  interoperability  and  integration  [DoD  DAG]. 
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3.  Airborne  M  ission  Systems  Bayesian  N etwork  M  odei 

3.1  Topology  of  the  Model 

This  secti  on  concentrates  on  thefi  rst  step  of  constructi  ng  a  BN  -  bui  I  di  ng  thetopol  ogy  of  a  BN 
which  represents  how  the  dependabiiity  factors  are  interconnected  with  the  architecture 
quai  ity  criteri  a.  We  wi  i  i  i  dentify  the  major  quai  ity  factors  and  archi  tecturecriteri  a  (attri  butes) 
contri  buti ng  to  the  dependabi  i  i ty  factors. 

OurBayesianN  etw  orkcaiiedAMS-BN  (AirMission  System  -  Bay  esi  an  N  etw  or  k)  f  oi  i  ow  s  th  e 
McCaii  quaiitymodei  [McCaii  1994]. Theattri butes comprisingqua//fycr/fer/aarebroken  into 
two  iayers:  Levei  1  and  Levei  2.  The  purpose  is  to  estabiish  a  hierarchy  of  architecture 
attri  butes  (characteristics),  so  that  morecompiex  attri  butes  can  be  broken  down  into  simpier 
measurabie  attri  butes  or  attri  butes  that  can  be  assessed  against  a  checkiist.  Thehierarchy  is 
shown  in  Figure 2. 

The  vari  abi  es  represent!  ng  the  quai  ity  factors  (the  user  poi  nt  of  vi  ew)  are  given  bei  ow .  i  n  the 
most  si  mpii  Stic  case,  we  capture  them  with  a  range  of  discrete  vaiues  (high,  medium,  iow}. 

•  Reliability  {high,  medium,  Zoi/i/}-  continuity  of  correct  service; 

•  Safety  {high,  medium,  low}  -  absence  of  catastrophic  consequences  on  the  users  or 
environment;  and 

•  M  aintainability  {high,  medium,  low}  -  abi i ity  to  undergo  modifications  and  repairs. 

Architecture  evaiuation  often  accounts  oniy  for  the  architecture  characteristics,  i.e.  the 
techni  cai  aspects.  Consi  deri  ng  that  the  maturity  of  the  engi  neeri  ng  process  and  the  previ  ous 
experience  of  an  organisation  have  significant  impact  on  the  outcome  of  projects  in  the 
avionicsdomain,wedefineand  add  attri  butes  characterisingtheorganisati  on. Thuswearrive 
at  a  set  of  software  architecture  attri  butes  associated  with  the  ubiquitous  triangie  -  Peopie, 
Process  and  Technoiogy  (Architecture). 


Figure2:  AMS  quality  measurement  framework 
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3.1.1  Level  1  Criteria  (Attributes) 

The  following  list  represents  theLevel  1  quality  criteria  related  to  the  dependability  factors. 

The  causal  relationships  between  the  criteria  and  thechosen  attributesareshown  in  Table! 

1.  0  rganisational  capability  {high,  low}  -  refi ects  the  I evel  of  maturity  of  the  organisati on 
and  the  project. 

2.  M  odularity:  {high,  Zoi/i/}.  This  isthedegreeto  which  a  system  or  program  is  composed  of 
discrete  components,  such  that  a  change  to  one  component  has  minimal  impact  on 
other  components  [IEEE-100].  A  highly  modular  architecture  enables  upgrades  and 
selective  replacement  of  modules  at  a  lower  cost.  It  will  provide  for  better  error 
containment  and  will  allow  gradual  system  integration. 

3.  Complexity  {high,  low}.  Thiscriterion  ref  I  ects  whether  the  architecture  is  perceived  as 
compi  ex  [ G u rp  2003] .  I  ncrease  i  n  compi  exi ty  w i  1 1  negati  vel y  i  mpact  al  I  d epend abi  I  i ty 
attributes.  Correlations  between  complexity  and  errors  and  between  complexity  and 
difficulty  to  understand  software  are  established  in  [Watson  etal.  1996].  Considering 
that  morecompi ex softwareismoredifficuitto  understand  and  that exhaustivetesting 
may  not  be  possible  (complexity  makes  software  testing  harder),  a  high  degree  of 
compi  exity  wi  1 1  ulti  mately  i  mpact  on  the  rel  i  abi  I  ity  of  a  system.  As  poi  nted  at  [Watson 
etal.  1996],  for  a  fixed  level  of  effort,  complexity  measures  reliability  itself. 

4.  Modifiability  {high,  low}  -  This  criterion  reflects  the  ability  of  the  architecture  to 
accommodate  changes  -  upgrades,  substitutes,  replacements. 

5.  Scalability  {high,  /oi/i/}-Theability  to  providefunctionality  up  and  down  agraduated 
series  of  application  platforms  that  differ  in  speed  and  capacity  [IEEE-100],  i.e.  this 
criterion  reflects  the  ability  of  the  system/  architecture  to  increase  its  performance 
when  new  hardware  or  software  components  are  installed. 

6.  Openness  {yes,  no}- Open  architectureissuch  architecturefor  which  design  parameters 
and  specifi  cati  ons  are  made  avai  I  abi  e  to  any  and  al  I  vendors  or  manufacturi  ng  fi  rms, 
thus  encouraging  development  of  compatible  products  and  enhancements. 

7.  Fault  tolerance  {high,  medium,  low}  -  This  characterises  the  ability  of  a  system  or 
component  to  conti  nu  e  normal  operati  on  d  espi  tethe  presence  of  hard  w  are  or  softw  are 
faults.  It  is  based  on  Error  Detection  and  System  Recovery  (rollback,  roll  forward, 
compensation,diagnosis,  isolation,  reconfiguration  and  re-initialisation).Thechoiceof 
error  detection,  error  and  fault  handling  depends  on  the  fault  assumptions.  The 
criteri  on  takes  i  nto  account  the  i  mpl  ementati  on  of  di  agnosti  c  procedures,  bui  It-i  n  tests, 
support  for  graceful  degradation  following  specified  priority  and  ongoing 
management  of  the  system  health. 

8.  Robustness  {high,  medium,  low}  -  Robustness  is  the  degree  to  which  a  system  or 
component  can  function  correctly  in  the  presence  of  invalid  inputs  or  stressful 
environmental  conditions  [IEEE-100].  We  need  to  view  robustness  not  only  as  a 
characteristic  of  a  standalone  system,  but  to  anticipate  that  the  modern  AMS  is 
designated  to  act  within  a  system-of-systems. 

9.  Testability  {high,  medium,  low}  -  The  degree  to  which  the  design  of  a  system  or 
component  facilitates  the  establishment  of  test  criteria  and  test  execution. 
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Tablelrepresentsthemapping  of  Level  1  quality  attributes  to  quality  factors.  Thetableshows 
only  quality  attributes  that  are  directly  connected  to  quality  factors.  It  should  be  noted  that 
some  attri  butes  i  nf  I  u  ence  d  i  fferent  factors  i  n  opposi  te  d  i  recti  on ;  for  exampi  e  i  mpl  ementi  ng  a 
certain  degree  of  fault  tolerance  improves  reliability,  while  at  the  same  time  increases 
complexity,  which  in  turn  negatively  impacts  testability  and  reliability. 

Table  1:  Qu  all  ty  F  actors  vs.  C  ri  ter  I  a 


Reliability 

Maintainability 

Safety 

Modifiability 

X 

Modularity 

X 

X 

X 

Openness 

X 

Organisational  capability 

X 

X 

X 

Robustness 

X 

X 

Scalability 

X 

Testability 

X 

X 

X 

Fauittolerance 

X 

3.1.2  People  Related  Level  2  Attri  butes 

Most  of  the  technologies  and  thetheories  of  software  engineering  are  human  based,  and,  as 
such,  depend  on  the  vari  ati  ons  of  ski  1 1 1  evel ,  competence  and  moti  vati  on .  The  provenance  of 
opensourcesoftwareisoneexampleof  peoplerelated  aspectsthat  may  be  considered  during 
B/aluation.  In  addition  to  the  developer/  manufacturer  issues,  there  are  human  factors  and 
concerns  on  the  user/ operator  side,  where  the  operator  can  be  viewed  as  an  integral 
component  of  the  system  or  as  an  enti  ty  outsi  d  e  of  the  system.  Of  al  I  possi  bl  e  cri  teri  a  w  e  have 
selected  only  three  which  are  expected  to  capture  the  competency  of  the  organisation  and 
developers. 

1.  Experience  within  the  domain  -  reflects  the  level  of  experience  the  development 
organ!  sati  on  has  w  i  th  the  pi  atform  and/  or  w  i  th  a  gi  ven  appi  i  cati  on .  -{L ow  (experi  ence 
less  than  2years),  Nominal  (2-5years),  and  High  (more  than  Syears)}. 

2.  Skill  retention  -  reflects  how  well  the  skills  are  retained  on  the  project.  We  use  the 
annual  turnover  rate  to  characterise  this  variable:  -{H  igh  retention  (turnover  rate  of  5- 
6%  or  less).  Low  (turnover  rate  of  15-20%  and  more)}. 

3.  M  anagement  practices  -  project  management  approachescan  increasethetechnical  risk 
for  a  project  [Goldsmith  et  al.  2008].  Risk  tolerance,  skill  and  experience  of  the 
management  team  will  impacttheoutcomeof  a  project.  We  apply  two  ratings:  (Low 
(poor)  and  High  (good)}. 

3.1.3  Process  Related  Level  2  Attri  butes 

I.  Process  maturity 

The  i  mpact  a  process  maturity  has  on  thedependabi  I  ity  can  be  i  1 1  ustrated  by  the  phase 
containment  effectiveness  and  customer  reported  defects  as  shown  inTable2[Diazet 
al.2002]. 
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Wewill  use  the  Capability  Maturity  Model  Integration  (CM  Ml)  maturity  level  asa 
quantitative  measure  of  the  quality  of  the  process  within  a  project.  We  can  use  the 
resu  I  ts  f rom  ap p rai  sal  an d  sel  f  assessment  as  evi  d  en ce  of  th e  matu  r i  ty  I  evel ,  or  w  e  can 
"assign"  a  level  based  on  analysis  of  the  available  project  plans,  development 
docu  mentati  on,  project  re/i  ews  and  mi  nutes.  The  process  matu  ri  ty  vari  abl  e  may  have 
oneof  the  foil  owing  states:  ■{Level_3_5  and  Level_l_2}. 

Other  equivalent  processor  quality  standard  would  beTickIt,  SPICE,  and  ISO  9002. 
They  definedocumentation  and  actions  required  to  deliver  quality  software. 

Table2:  General  Dynamics  D  ecision  Systems  Project  Performance  versus  CM  M  levd 


CMM 

Level 

Percent 

Rework 

Phase 

Containment 

Effectiveness 

C  ustomer  Reported 
Unique  Defects 
Density  perKLOC 

Productivity 
(X  Factor 
Relative) 

2 

23.2% 

25.5% 

3.20 

lx 

3 

14.3% 

41.5% 

0.90 

2x 

4 

9.5% 

62.3% 

0.22 

1.9x 

5 

6.8% 

87.3% 

0.19 

2.9x 

2.  R  devan t  standards 

Standards  such  as  RTCA  DO-178  (RTCA  DO-278),  the  Software  Development 
Standard  for  Space  Systems  (SDSSS),  IEEE  982.2  "IEEE  Guide  for  the  Use  of  IEEE 
Standard  Dictionary  of  Measures  to  ProduceReliableSoftware",  Def(Aust)  5679  etc. 
i  mpact  dependabi  I  i  ty,  especi  al  I  y  i  f  used  w  i  thi  n  a  matu  re  organ  i  sati  on .  The  appi  i  cat!  on 
of  a  relevant  standard  would  decrease  the  number  of  defects  in  code  and 
documentation  and  thus  improve  reliability,  aval  I  ability  and  safety.  In  order  to  come 
up  with  a  concrete  number  we  used  a  CoComo  based  tool  (CostXpert)  to  provide  a 
crude  guideline.  The  tool  is  providing  an  estimation  of  the  number  of  defects  for 
selected  typeofprojects(embedded,  military  etc),  lifecycleand  applied  standards.  We 
realise  that  selecting  the  type  of  project  may  define  quite  a  loose  framework,  but 
experimenting  with  different  combination  we  assume  that  the  application  of  an 
appropriate  standard  would  translate  into  a  12-15%  decrease  of  defects  (in  code  or 
documentation).  The  vari  able  has  two  states  Ves,  no}. 

3.1.4  Technology/  Product  Related  Level  2  Attributes 
I.  Architecture  style/patterns 

The  architectural  patterns  are  one  of  the  techniques  for  designing  high  quality 
software  architectures.  As  pointed  in  [Booch  2006]  we  are  beginning  to  observe  the 
emergence  of  domain-specific  software  architectures.  They  represent  reusable 
solutionswhich  evol  ved  w  i  th  i  n  d  i  fferent  appi  i  cati  on  d  omai  ns.  D  i  fferent  patterns  will 
have  a  different  impact  on  the  reliability,  maintainability  and  performance  of  the 
architecture.  Based  on  practical  experience,  [Buschman  et  al.  19%]  grouped  the 
architecturestructures  in  several  cl  asses  (patterns),  namely.  Layered,  Pipes  and  Filters, 
Blackboard,  Distributed  Systems  Broker,  Model -View-Controller,  Presentation- 
Abstraction-Control  and  Microkernel.  Wewill  refer  the  reader  to  the  book  for  further 
detailson  these  patterns,  thecontextthey  are  most  suitablefor,  and  how  they  relateto 
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thesoftwareprocess and  various softwaredevdopment techniques.  Weadopted  this 
ciassifi cation  scheme  because  it  is  practicai  and  can  cover  most  of  the  existing 
architectures  under  investigation. 

Svanberg  and  Wohiin  interviewed  a  group  of  software  architects  and  captured  the 
data  for  a  consequent  anaiysis  and  comparison  between  software  architectures  with 
respect  to  sdected  quaiity  attributes:  efficiency,  functionaiity,  usabiiity,  rdiabiiity, 
mai  ntai  nabi  i  ity  and  portabi  i  ity  [Svahnberg  et  ai .  2005].  A  ithough  the mai  n  concern  of 
this  work  wasthedevdopment  of  a  methodoiogy  for  di  citing  the  views  of  different 
stakehoiders,  thereby  formui ati ng  a  framework  for  quai ity  attri  butes  and  anaiysis  of 
d i fferent  arch i tectu rai  stru ctu res,  w e  consi  d er  the  d ata  captu red  by  the  i  ntervi  ews  as 
soi  i  d  qu anti  tati  ve  expert  assessment  w  h i  ch  can  be  used  to  f  i  i  i  i  n  the  N  od e  P robabi  i  i  ty 
Tabiefor  the  variabie  "Architecture  styie".  According  to  the  study,  the  ranking  of 
archi  tectu  re  structu  res  per  qu  ai  i  ty  attri  bute  i  s  presented  i  n  Tabi  e  3  ( w  e  qu  ote  on  i  y  the 
attri  butes  of  i  nterestto  us).  Onesuggesti  on  offered  by  the  report  is  that  no  architecture 
is  reaiiy  strong  in,  or,  isfocused  on,  some  specific  quaiity  attri  butes  (rdiabiiity  bdng 
one  of  them).  Consequentiy,  we  give  rdativdy  iow  wdght  to  the  variabie 
"Archi  tectu  re  styie". 

T ableS:  Framework  for  quality  attributes  as  in  [Svahnberg  et  ai.  2005] 


Quality 

Attribute 

M  icrokernel 

Blackboard 

Layered 

Model-View- 

Controller 

Pipes  and 
Filters 

Functionality 

0.228 

0.261 

0.179 

0.188 

0.144 

Reliability 

0.207 

0.103 

0.270 

0.239 

0.180 

Maintainability 

0.124 

0.171 

0.283 

0.198 

0.225 

Portability 

0.194 

0.0767 

0.366 

0.157 

0.207 

2.  Quality  of  requirements  elicitation:  {high,  low}.  Someofthecausesforprojectfaiiurestem 
from  this  phase  of  the  devdopment  cycie,  \.e.  Ad  hoc  requirements  management, 
ambiguous  and  imprecise  communication  and  undetected  inconsistencies  in 
requirements,  designs,  and  impiementations. 

3.  Cohesion  (strong,  weak}  -  The  manner  and  degree  to  which  the  tasks  performed  by  a 
si ngie software  moduieare  rdated  to  one  another  [iEEE-100]. 

4.  Coupling  ^ight,  loose}- Themanner  and  degreeof  interdependencebetween  software 
moduies.  Aspects  to  be  considered  may  be  dependencies  on  common  environment, 
content  cou pi ing,  controi  coupiing,  data  coupiing  etc. 

5.  Change  propagation/containment  {yes,  no}  -  functionaiity  is  mapped  to 
components/  moduiesinsuchawaythatachange/  upgradeinfunctionaiity  wiii  affect 
oniy  known  and  isoiated  components  and  not  affect  others  impiementing  different 
functionaiity. 

6.  Error  propagation/containment  (high,  low}  -  Error  propagation  from  one 
component/ moduie  to  another  refiects  the  iikeiihood  that  an  error  in  the  first 
component  wiii  cause  an  error  in  the  second  component  (considering  that  the  first 
component  feeds  i  nformati  on  to  thesecond).  Low  vai  uefor  this  attri  bute  means  that  a 
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failure  in  any  component/  module  will  not  result  in  unintended  failures  in  other 
system  modules. 

7.  Views  documentation  {documented,  not-documented}-  reflectsthequality  of  documenting 
the  architecture.  The  various  high-level  aspects  of  software  architecture  are  often 
called  views:  "A  view  represents  a  partial  aspect  of  asoftwarearchitecturethatshows 
specific  properties  of  a  software  system"  [Buschman  etal.  19%].  [SWEBOK  2004]  lists 
different  sets  of  views  that  have  been  suggested;  the  evaluation  of  this  attribute, 
however,  should  be  concerned  with  how  complete,  clear  and  comprehensive  is  the 
documentation  and  if  it  captures  well  the  requirements  and  elaborates  them  for  the 
next  stage  of  the  software  development  cycle. 

8.  Capacity  {high,  nominai,  iow}  -  The  ability  to  add  new  functionality  or  extend  the 
existing  one  will  depend  on  the  resource  usage  such  as  CPU  (peak  CPU  load,  spare 
capacity),  power  consumption,  1-0  rates  (e.g.  expected  network  usage),  the  processor 
load,  memory/  HDD  utilisation,  network  loads. 

9.  Interoperabiiity  {yes,  no}  -  This  criterion  defines  the  degree  to  which  information  or 
services  can  be  exchanged  to  enable  them  to  operate  effectively  together.  The 
requi  rementsfor  i  nteroperabi  I  ity  wi  1 1  i  mpactthescalabi  I  ity,  compi  exity  and  ulti  mately 
the  reliability  and  maintainability  of  the  software  architecture. 

10.  HW-isoiation  {yes,  no}-  reflects  the  degree  to  which  the  application  layer  is  isolated 
from  the  hardware  specifics.  This  attribute  covers  questions  such  as:  (1)  is  there  a 
hardware  abstraction  layer;  (2)  do  applications  use  low-level  constructs  (e.g.  no 
assembler)  or  access  hardware  directly;  (3)  does  the  software  rely  on  graphics 
accelerators  that  may  inhibit  portability  etc. 

11.  Scheduiing  {deterministic,  non-deterministic}-  the  scheduling  algorithm  in  embedded 
systems  will  affect  future  upgrades.  For  example,  rate-monotonic  scheduling  gives 
determini  Stic  guarantees  with  regards  to  response  times. 

12.  /  n  ter  faces  {open,  propri  etary  /-This  attri  bute  ref  I  ects  whetherwelldefined,widelyused, 
non-proprietary  interfaces  and  protocols  are  used.  In  relation  to  networks,  it  reflects 
whether  open  network  standards  and  COTS  components  are  used. 

13.  Technoiogy  independence  {yes,  no}  -  This  attribute  reflects  the  extent  to  which  the 
architecture  depends  on  current  technology,  e.g.  application  of  standards  and 
established  COTS  products,  availability  of  DM  SMS  plans  and  technologly  refresh 
plans,  identification  of  coding  standards  etc. 

14.  Fauit  pre/ention  {yes,  no}  -  aims  to  avoid  fault  occurrences  by  construction,  i.e. 
information  hiding,  usage  of  strongly-typed  programming  languages  etc. 

15.  M  emory  management  (good,  bad}  -  the  attribute  reflects  whether  there  is  dynamic  or 
stati  c  al  I  ocati  on  of  memory. 

16.  Task  management  {controiied,  dynamic}  -  reflects  specifics  of  task  management  such  as 
how  the  application  manages  tasks,  creation  of  dynamic  tasks,  deletion  of  tasks, 
assi  gnment  of  task  pri  oriti  es. 
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3.2  Dependencies  Definition 

Each  node  in  a  Bayesian  Network  is  associated  with  aCPT  which  maps  the  probabilities  of  a 
node  to  each  configuration  of  parent  values. 

For  root  nodes theCPT  represents  basic  knowledge(or  opinion,  beliefs).  TheCPT  tablefor 
root  nodes  is  assigned  values  based  on  knowledge  about  a  specific  architecture  which  is 
gai  n ed  as  a  resu  1 1  of  arch  i  tectu  re  revi  ew s,  au d  i  ts,  an al  ysi  s  of  aval  I  abl  e  d  ocu  mentati  on  etc.  To 
hel  p  w i th  thi s  process  a  checki  i  st  i s  associ  ated  w i th  every  root  node.  The checki  i  st  i s  expected 
to  help  the  user  and  is  at  the  same  time  a  tool  to  capture  the  knowledge  of  experts  in  the 
relevant  field.  Wewill  not  go  through  all  nodes,  butjustto  demonstratethe  logic  we  follow 
whilefilling  in  CPT,  wewill  just  givean  exampleof  theCPT  and  checklist  associ  ated  with  one 
of  the  root  nodes  (attribute)  and  one  intermediate  node  (criteria). 

The  checklist  shown  in  Table  4  is  based  on  [Goldsmith  et  al.  2008]  and  lists  a  number  of 
concerns  when  analysing  the  node 'M  anagement  Practices'.  It  is  expected  that  the  user  of  the 
mod  el  w  i  1 1  make  an  assessment  usi  ng  such  checki  i  sts  i  n  ad  d  i  ti  on  to  other  rel  evant  i  nformati  on 
before  enter!  ng  val  ues  i  n  the  rel  evant  CPT. 


T able  4:  M  anagement  practices  attribute  -  checklist  of  concerns  to  be  addressed 


Quality  Criteria 

Checklists 

M  anagement  practices 

1.  1  s there i ncreasi ng conti ngency(funding and  sched u  1  e conti ngency i ncrease 
acceptance  of  risk) 

2.  Compromising  engineering  rigour  in  order  to  meet  schedule  cost 

3.  Del  egati  on/  assi  gnment  ri  sk  to  a  contractor  or  another  project 

4.  Is  there  deferred  effort  (deferred  effort  tends  to  realise  technical  risks). 
Typical  violations  are  writing  design  documents  post  coding  effort, 
retrofitting  certification  evidence  after  the  system  is  developed  etc. 

5.  Is  the  risk  mitigation  activity  resourced  properly 

6.  1  s  the  esti  mati  on  based  on  real  i  sti  c  assu  mpti  ons  or  i  s  i  t  a  best  case  esti  mate 
(e.g.  assuming  the  participation  of  "A"  team) 

7 .  Is  key  system  anal  ysi  s  compi  eted  before  start!  ng 

Two  more  examples  are  the  CPT  for 'Skill  retention'  and  'Process  maturity'. 

Skill  retention:  According  to  [Roman  2007]  the  average  annual  turnover  rate  for  the 
semi  cond  uctor  i  nd  ustry  i  s  7%  (bei  ng  7.5%  i  n  the  past  few  years)  and  rang!  ng  from  high  12%  to 
less  than  1%.  [AEA  1995]  showsthattheturnoverratefor  all  engineers  in  the  Electronics  and 
I  nformati  on  Technology  companies  was  11.4%  compared  with  9.5%  in  1993  and  9.7%  in  1992. 
For  1998  AEA  reported  an  average  turnover  rate  of  21.8%  for  electronics  engineers  at  its 
member  companies,  but  25.5%  for  software  and  programmer  analysts  and  23. 7%  for  software 
engi  neers.  Based  on  these  figures,  wedefi  ned  our  expectation  for  theattri  bute'Ski  1 1  retention'. 

Process  maturity:  According  to  [CM  M  |_SEI  2006],  out  of  1377  report!  ng  organisations  18.2%  are 
at  Level  5, 4.4%  are  at  I  evel  4, 33.8%  are  at  I  evel  3, 33.3%  are  at  I  evel  2, 1.9%  are  at  I  evel  1  and 
8.4%did  not  respond.  Wecan  usethisdatato  initially  definetheCPT  -38.24%  probability  for 
Level_l_2  and  61.76%  probabi  I  ity  for  Level_3_5.  Consider!  ng  that  the  reported  prof!  le  is  not 
d  rasti  cal  I  y  d  ifferent  from  a  si  mi  I  ar  one  prod  uced  I  n  2004,  the  assu  mpti  on  shou  I  d  be  real  I  sti  c. 
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When  wehaveaspecific  project  in  mind,  then  wewould  be abie to  assign  100%probabiiityto 
one  of  the  vai  ues  of  'process  maturity'  and  0%  to  the  other  vai  ue. 

CPT  for  the  node  'Organisation  capabiiity'  is  shown  in  Tabie  5.  The  numbers  refiect  our 
opinion  which  is  derived  from  experience  and  reievant  pubiications.  According  to  [Jones 
2000],  the  experience  i  eveis  of  both  the  managers  and  the  techni  cai  staff  i  n  bui  i  di  ng  si  mi  iar 
type  of  appiications  has  a  combined  impact  on  productivity  of  120%,  whiie  effective 
methods/  processes  have  a  positive  impact  of  oniy  35%.  Thefoiiowing  reverse  effect  isaiso 
cited:  among  the  factors  that  can  reduce  or  degradesoftware  productivity,  management  and 
staff  inexperience  have  a  combined  negative  impact  of  177%,  whiie  ineffective 
methods/  processes  negative  impact  is41%.  Based  on  this,  wewiii  attribute  more  weight  to 
the'Skiii  retention'  and  'Management  practices',  than  'Process maturity'. 


Tables:  CPT  for  thenode  'Organisation  capability’ 


Process 

Skill/  retention 

M  anagement 
practices 

Architecture 

quality 

Organisation 
capability  H 

Organisation 
capability  L 

La/el_3_5 

Nominal 

High 

High 

90% 

10% 

La/el_3_5 

Nominal 

High 

Low 

70% 

30% 

Level_3_5 

Nominal 

Low 

High 

75% 

25% 

Level_3_5 

Nominal 

Low 

Low 

30% 

70% 

Level_3_5 

Low 

High 

High 

85% 

15% 

La/el_3_5 

Low 

High 

Low 

65% 

35% 

La/el_3_5 

Low 

Low 

High 

70% 

30% 

La/el_3_5 

Low 

Low 

Low 

25% 

75% 

La/el_l_2 

Nominal 

High 

High 

65% 

35% 

Level_l_2 

Nominal 

High 

Low 

35% 

65% 

La/el_l_2 

Nominal 

Low 

High 

40% 

60% 

La/el_l_2 

Nominal 

Low 

Low 

15% 

85% 

Level_l_2 

Low 

High 

High 

60% 

40% 

La/el_l_2 

Low 

High 

Low 

30% 

70% 

Level_l_2 

Low 

Low 

High 

35% 

65% 

La/el_l_2 

Low 

Low 

Low 

10% 

90% 

3.3  Application  of  the  Airborne  M  ission  Systems  BN  M  odel 

The  probabi  i  i  ti  es  generated  by  the  modei  are  expected  to  be  i  nterpreted  as  a  gu  i  dei  i  ne,  i  .e.  we 
are  iooking  for  the  trend  whether  specific  probabiiity  is  high  or  iow.  Based  on  this,  a 
conciusion  can  be  made  about  the  associated  risk.  The  resuit  of  the  architecture  evaiuation 
usingtheAMS_BN  modei  wiii  depend  on  thequaiity  of theassessmentoftheroot nodes. The 
expert  needs  to  consider  the  existing  evidence  and  define  the  CPT  for  the  root  nodes,  in  the 
absence  of  enough  evidence  aboutaspecificroot  node,  adecisi  on  needsto  bemadewhether 
to  (1)  usethedefauitCPT  (which  aims  to  represent  industry  averages),  or  (2)  defineCPT  with 
more  conservative  numbers,  or  (3)  re-consider  the  impact  of  this  root  node  on  itschiidren. 
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4.  Conclusions  and  Future  Work 

At  the  phase  of  architecture  definition  a  direct  evaluation  of  dependability  is  not  possible, 
however,  some  i  nformati  on  is  al  ready  aval  I  abl  e,  e.g.  the  qual  i  ty  of  the  software  dev  el  opment 
process,  software  standards  applied,  project  environment,  analysis  of  requirement 
specifications,  architecture  level  models,  expert  opinion  etc.  This  evidence  is  uncertain  and 
incomplete:  noneof  it,  by  itself,  can  provide  enough  information  aboutthequality  of  software. 
Therefore,  in  order  to  build  a  holistic  pictureof  the  state  of  dependability,  a  type  of  causal 
model  -  Bayesian  Network  model,  is  proposed.  The  AMS-BN  model  considers  evidence 
generated  up  to  (and  including)  the  architecture  definition  phase.  The  topology  and  the 
dependencies  definitions  of  the  AMS-BN  are  described.  This  work  demonstrates  the 
application  of  BN  approach  intheAirborneMission  Systemsdomain. The  created  model  is 
ai  med  to  assi  st  the  A  M  S  team  w  i  th  thetechni  cal  ri  sk  assessment  of  mi  ssi  on  system  software  i  n 
Defence  acquisition  projects.  Netica  [Netica  RM  2007]  is  used  to  run  the  model. 

The  AMS-BN  model  is  currently  being  applied  to  a  Defence  project,  and,  when  the  project  is 
completed,  the  model  prediction  will  be  compared  with  the  project  outcomes.  This  will  be 
used  as  a  validation  approach,  in  addition  to  peer  reviewsand  checking  that  theresults  match 
common  sense  and  observed  results  from  previous  projects. 
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Figures:  AM5-BN  connectivity 
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